OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability#3918
OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability#3918marcschier wants to merge 67 commits into
Conversation
…ng with hardened shared store Adds an opt-in provider model (Opc.Ua.Server.Distributed) to replicate address-space topology/values and session state across server replicas for active/passive and active/active HA, exposed via documented OPC UA redundancy (ServiceLevel/RedundantServerArray). The single-instance in-memory path stays zero-overhead (default NullRecordProtector). Building blocks: ISharedKeyValueStore (+in-memory), INodeStateStore, address-space synchronizer, leader election (static + shared-store lease CAS), service-level providers, distributed value cache, shared session store. Wired via IServerStartupTask hosting seam + fluent DI (UseDistributedAddressSpace / AddServerServiceLevel / redundancy options). CustomNodeManager2 opts in via ILocalAddressSpaceSource. Security hardening (plan 30, SDL + IEC 62443): IRecordProtector + AesCbcHmacRecordProtector (AES-256-CBC Encrypt-then-MAC, verify-before-decrypt, fail-closed) on every shared record; KeyRingRecordProtector (staged key rotation); SharedSingleUseNonceRegistry (cross-replica single-use server nonce via store CAS); key zeroization. Docs/HighAvailabilitySecurity.md captures the STRIDE threat model and operator guidance. Docs: HighAvailability.md, HighAvailabilitySecurity.md, Docs/README link, HighAvailabilityServer sample. Plans 28/29/30. 85 Distributed unit tests (net10.0 + net48).
…reconnect (S5) Implements the opt-in mirrored fast-reconnect from plans 30/31. After a failover a client reconnects to a standby by re-running ActivateSession; the AuthenticationToken is only a lookup key and the standby still performs the full client-certificate signature validation against a mirrored, single-use serverNonce (token-only hijack and nonce replay are both closed). - SharedSessionEntry: extended with the full reconstruction state (serverNonce, clientNonce, client cert blob, SecurityPolicy/Mode, endpoint, timeout, client description); encrypted at rest via the record protector. - SessionManager: additive, backward-compatible RestoreSessionAsync + SupportsSessionRestore seam in the ActivateSession miss-path (default returns null => unchanged behaviour; m_sessions stays private). - DistributedSessionManager: mirrors encrypted session state on create/activate, removes on close; on restore enforces REQ-UA-7 (same SecurityPolicy/Mode), consumes the serverNonce single-use across the replica set (replay defence), reconstructs the session, and logs provenance with a one-way token digest. - ISessionManagerFactory seam on StandardServer (wired from DI by the hosted service, supplying a server-certificate provider) + DistributedSessionManagerFactory + UseDistributedSessions(...) fluent API. Safe default EnableFastReconnect=false (re-auth on failover). - Docs (HighAvailability.md, HighAvailabilitySecurity.md) updated. Tests: 97 Distributed (net10+net48); no regression (61 Session + 57 client SessionTests integration pass). Remaining: two-server network e2e (S6).
…roring (S6) DistributedSessionMirrorIntegrationTests stands up a fully-started server whose ISessionManagerFactory builds a DistributedSessionManager, and verifies end-to-end that a session created/activated through the real service handlers is mirrored encrypted to the shared store (a wrong-key reader fails closed) and removed on close. This closes the factory -> StandardServer.CreateSessionManager -> mirror runtime-wiring gap. The full secured two-server token-reuse reconnect happy-path remains a documented follow-up (the stack client does re-auth-on-failover; the direct-service helper only drives unsecured sessions). The restore decision logic (REQ-UA-7 + single-use nonce/replay) is unit-tested and the base ActivateSession signature path is integration-tested. 98 Distributed tests pass (net10 + net48).
…t (F6/F9) Workstream A from plans/32: - F6: SharedKeyValueSessionStore keys entries by the SHA-256 digest of the AuthenticationToken (SharedKeyValueSessionStore.KeyFor) instead of the raw token, so a backend's key enumeration/monitoring/dumps never expose the token. Test asserts the keyspace contains no raw token. - F9: a successful cross-replica restore emits a distinct AuditSessionEventState (Session/RestoredFromSharedStore) via the new IAuditEventServer.ReportAuditSessionRestoredEvent, in addition to the standard AuditActivateSession, with a one-way token digest for provenance. - F7: analyzed — the decrypted serverNonce becomes the session's retained working Nonce (no extra plaintext copy in the manager); Nonce.Data zeroization on dispose is a pre-existing server-wide Core concern tracked separately. Docs updated. 99 Distributed tests pass (net10 + net48).
…ng in sample (FG) Workstream FG from plans/32: - New Docs/KubernetesDeployment.md: worked replicaset deployment (StatefulSet + headless Service, leader election, readiness tied to ServiceLevel, KEK + shared ApplicationInstanceCertificate provisioning via Secrets, security checklist). Linked from Docs/README.md and HighAvailability.md. - HighAvailabilityServer sample: wires UseDistributedSessions (opt-in HA_FAST_RECONNECT) and an optional AesCbcHmacRecordProtector from a base64 HA_RECORD_KEY (encrypted shared store), demonstrating the production-hygiene pattern. - Transparent redundancy remains a documented deployment pattern (single virtual endpoint + shared session store + subscription transfer; no new transport).
…se failover) design
…A-13, B) On failover to a redundant server, when EnableTokenReuseFailover is set, the client re-activates the existing session by reusing the current AuthenticationToken (signing over the new channel + last serverNonce) instead of CreateSession, falling back to re-authentication if the standby rejects the token. - Session.cs: extracted ReactivateExistingSessionAsync (the token-reuse activation core, shared with UpdateSessionAsync); RecreateInPlaceCoreAsync tries it first (adopting the failover server's cert) before the existing clear + fresh-CreateSession fallback; new EnableTokenReuseFailover property (copied across recreate clones). OpenAsync is untouched. - ManagedSession: EnableTokenReuseFailover option threaded through CreateAsync + ctor and applied to the inner session; ManagedSessionOptions.EnableTokenReuseFailover; ManagedSessionBuilder.WithTokenReuseFailover(). Default off (re-auth on failover). No regression: 57 client SessionTests + 135 reconnect/failover integration tests pass (net10). Updated the ctor-reflection test for the new parameter.
DistributedSessionFailoverIntegrationTests: two secured servers share one store via DistributedSessionManager; a ManagedSession client with WithTokenReuseFailover fails over from the active to the standby. The standby restores the mirrored session and re-activates it with the reused AuthenticationToken, so the client's SessionId is preserved (a fresh re-authentication would change it). Passes on net10 + net48. Docs (HighAvailability.md, HighAvailabilitySecurity.md) updated: client WithTokenReuseFailover() usage + the e2e validation; plans/32 marks B and C done.
Split all distributed/HA implementation into a new OPCFoundation.NetStandard.Opc.Ua.Server.Distributed project that references Opc.Ua.Server; core now keeps only the seams. - Move ILocalAddressSpace/ILocalAddressSpaceSource to the Opc.Ua.Server namespace (NodeManager/); extract a shared internal PredefinedNodesAddressSpace reused by both CustomNodeManager2 and AsyncCustomNodeManager (both now implement ILocalAddressSpaceSource). Apply path is async (no sync-over-async). - Remove the node-state-store-registry hook from ServerInternalData; the distributed startup task owns its own registry. - Add CryptoUtils.ZeroMemory/FixedTimeEquals polyfills and reuse them in the record protector, EncryptedSecret, and KeyCredentialBridgeAuthenticator. - Fix CA2213 in AddressSpaceSynchronizer (dispose enumerator in DisposeAsync). - Rebase HighAvailabilityServer sample onto AsyncCustomNodeManager; expand the sample README with shared-store wiring and an active/active setup. - Strip internal tracking tags (REQ-UA/Finding/IEC/SDL) from code and XML docs. - Merge HighAvailabilitySecurity.md into HighAvailability.md and fix links. - Wire the new project into UA.slnx, the test project, and the sample.
Add a separate net8.0+ package providing true active/active (multi-writer) replication for the distributed server, built on the Crdt + Crdt.Transport NuGet packages. Opt-in; the base distributed library and its active/passive path are unchanged. - New Libraries/Opc.Ua.Server.Distributed.Crdt project (net8.0;net9.0;net10.0, no-op shell on legacy CI legs via RestrictForLegacyTfm); references the base distributed project + Crdt + Crdt.Transport. - Address space A/A: CrdtAddressSpaceSynchronizer models topology and values as last-writer-wins maps and gossips state over Crdt.Transport (in-memory / TCP / UDP, optional TLS); every replica writes, received state is merged and applied. A topology merge preserves the locally-known value so it never regresses a concurrently-updated value (values are versioned by their own entries). - Sessions A/A: CrdtSharedKeyValueStore replicates encrypted session entries by gossip and reuses DistributedSessionManager; the single-use server nonce stays on a strongly-consistent ISingleUseNonceRegistry (CRDTs cannot enforce exactly-once), and the CRDT store rejects compare-and-swap. - Fluent opt-in: UseCrdtAddressSpace(...) / UseCrdtSessions(...) with a shared CrdtGossipOptions base (ReplicaId, UseTcpGossip/UseUdpGossip/AddPeer, limits). - Tests: new Opc.Ua.Server.Distributed.Crdt.Tests (net8/net10) — convergence, multi-writer, concurrent-value LWW, serializer round-trip, CAS/Watch boundary; plus a CRDT exercise in the AOT test project. - Docs: HighAvailability.md 'Active/active with CRDTs' section incl. the single-use-nonce security boundary; Crdt/Crdt.Transport added to CPM. Crdt/Crdt.Transport 1.0.0 (MIT, NativeAOT-ready) restore from nuget.org.
…mple, clarify HA docs - Rename the user-facing fluent surface from Crdt* to Replicated*: UseReplicatedAddressSpace / UseReplicatedSessions, ReplicatedAddressSpaceOptions / ReplicatedSessionOptions / ReplicatedGossipOptions, ReplicatedServerBuilderExtensions (implementation classes and the Crdt package/namespace keep their names). - Integrate the CRDT package into the HighAvailabilityServer sample with an HA_MODE switch (ap = active/passive leader-write, aa = active/active CRDT gossip via HA_GOSSIP_PORT / HA_GOSSIP_PEERS); document running two AA replicas in the sample README. - HighAvailability.md: reference the implemented CRDT library (not 'deferred'), document that a write to a non-leader replica is discarded in active/passive, and rewrite the value-participation note to describe how participation works. Addresses review comments on Docs/HighAvailability.md and the sample README.
…Ua.Server.Distributed.Tests Organize the source files of both Opc.Ua.Server.Distributed and Opc.Ua.Server.Distributed.Crdt into logical subfolders (AddressSpace, KeyValueStore, Redundancy, Values, Sessions, Security for the base library; AddressSpace, Sessions for the CRDT adapter), with the top-level DI entry points kept at the project root. Namespaces are unchanged. Mirror the organization in the test projects: - Extract the base distributed tests from Opc.Ua.Server.Tests/Distributed into a dedicated Opc.Ua.Server.Distributed.Tests project (mirroring the src project), with the same subfolders; add it to UA.slnx and InternalsVisibleTo. Test namespaces are unchanged. - Organize Opc.Ua.Server.Distributed.Crdt.Tests into AddressSpace/Sessions subfolders mirroring the CRDT src. Addresses review comments asking to mirror the test projects to the src side and to organize the files logically without changing namespaces.
Add unit tests for the previously-uncovered CRDT adapter surface flagged by codecov (patch coverage 67.65%): - ReplicatedGossipOptions/ReplicatedSessionOptions: defaults, UseTcp/UseUdp transport factory wiring, AddPeer, CreateReaderOptions/CreateTransport, null-arg guards. - CrdtAddressSpaceStartupTask: attaches a synchronizer to opted-in (ILocalAddressSpaceSource) node managers and skips the rest; null-arg guards. - CrdtSessionManagerFactory + UseReplicatedSessions/UseReplicatedAddressSpace registration; null-arg guards. - ByteStringCrdtSerializer JSON round-trip (null/empty/data); CrdtSharedKeyValueStore.ScanAsync prefix filtering. CRDT package line coverage 67% -> 93.7%; 25/25 tests pass on net8.0 and net10.0.
|
Addressed the codecov patch-coverage feedback in 4f4d7ed. The 80% patch gap was concentrated in the new
CRDT package line coverage rises from ~67% to 93.7%; 25/25 tests pass on net8.0 and net10.0. Codecov will re-evaluate the patch on this commit. |
…t + non-transparent Server model: per-mode Server.ServerRedundancy (RedundancySupport, RedundantServerArray, non-transparent ServerUriArray, transparent CurrentServerId); sub-range ServiceLevel providers (Table 105) + load balancing; RequestServerStateChange/Maintenance/EstimatedReturnTime (6.6.5, admin-gated); NTRS capability; FindServers returns the RedundantServerSet (IRedundantServerSetProvider). New shared Opc.Ua.ServiceLevels/ServiceLevelSubrange in core. Client: RedundantManagedClient realizing all Table 107 modes (Cold/Warm/Hot(a)/Hot(b)/HotAndMirrored); RedundancySupport wording; ServiceLevel sub-range failover rules; FindServers ServerUri->endpoint resolution (no security downgrade); client redundancy via TransferSubscriptions; non-transparent network redundancy. State mirroring (opt-in provider seams; single-instance unchanged): session takeover, subscription-definition mirror, async notification sequence/Republish mirror, best-effort continuation-point envelope, deterministic EventId provider, RegisterNodes replica-consistent. Extensions: Opc.Ua.Server.Distributed.Crdt (active/active) and Opc.Ua.Server.Distributed.Kubernetes (Lease election, peer discovery, ServiceLevel->readiness). Samples (HighAvailabilityServer, RedundantClient), docs mapped to 6.6, conformance tests. Security: AES-CBC+HMAC record protection, fail-closed RecordProtectionGuard for external stores (base + CRDT session paths), single-use server-nonce replay protection, K8s TLS hostname validation. Validated on net10 and net48.
… review findings CRDT active/active extension (Opc.Ua.Server.Distributed.Crdt): - Bump Crdt/Crdt.Transport 1.0.0 -> 1.0.2 (now ship netstandard2.0/2.1 assets) - Widen TFMs from net8.0+ to $(LibTargetFrameworks) (net472/net48/netstandard2.1/net8/9/10); remove RestrictForLegacyTfm; gate IsAotCompatible/binding-gen to net8+ - Fail closed for unauthenticated networked gossip (TCP without mutual TLS, UDP); add AllowUnauthenticatedGossip opt-out for dev/test; gossip-port NetworkPolicy guidance Server review findings: - AddServerRedundancy warns when non-transparent mode has no IServiceLevelProvider - Rename AddManualFailover -> AddRequestServerStateChange ([Obsolete] shim retained) - Consolidate ServerRedundancyOptions peer inputs (RedundantPeers canonical) - DeterministicEventId: exclude per-replica ReceiveTime, compute after distinguishing fields are populated (replica-stable per 6.6.2.2) - Subscription retransmission mirror: delta path, namespace/server tables once per subscription, bounded-parallel drain Client review findings: - Maintenance(0) fails over to a healthy peer (Table 105); EstimatedReturnTime backoff - Warm backups use MonitoringMode.Disabled until failover (Table 107) - Hot(b) dedup uses exact value identity (no hash-collision drops); value-struct key - Subscription-template ownership: client disposes retained templates - De-duplicate WithNetworkRedundancy registration Validated: 0 warnings; tests green on net10 + net48 (CRDT 30, Distributed 178, Client redundancy 75, InformationModel redundancy 7).
…eview feedback Rename (full identity: folders, csproj, AssemblyName, PackageId, RootNamespace, C# namespaces, references, UA.slnx, InternalsVisibleTo): - Opc.Ua.Server.Distributed -> Opc.Ua.Server.Redundancy - Opc.Ua.Server.Distributed.Crdt -> Opc.Ua.Server.Redundancy.Crdt - Opc.Ua.Server.Distributed.Kubernetes -> Opc.Ua.Server.Redundancy.K8s (identity only; the word "Kubernetes" and k8s client types are preserved in prose/code) - Tests mirror the rename (.Tests/.Crdt.Tests/.K8s.Tests/.Integration.Tests) - Application HighAvailabilityServer -> RedundantServer - Documentation updated to the new names (present tense, merged-to-master state) Review feedback (verified previously-resolved PR comments are satisfied; fixed gaps): - async node-manager base, A/A sample README, REQ-UA tags, CA2213, central CryptoUtils polyfills, ILocalAddressSpace namespace - all confirmed in current code Roadmap findings implemented: - OPCFoundation#28 ServerUriArray-only peers are now failover candidates (connect + read live level) - OPCFoundation#8 FetchRedundancyInfo follow-up reads merged into one ReadValuesAsync - OPCFoundation#29 ContinuationPoint mirroring documented as envelope-only (partial-SHALL boundary) - OPCFoundation#30 Server.ServerRedundancy typed to NonTransparent/Transparent subtype by mode - OPCFoundation#20 Add* (standard) vs Use* (extension) convention documented + ServiceLevel cross-ref - OPCFoundation#24 transparent-mode shared application-key blast-radius/rotation guidance expanded CI green (fixes pre-existing all-TFM build break, unrelated to the rename): - Opc.Ua.Sessions.Tests redundancy test doubles: implement IServerRedundancyHandler. ShouldFailover and use RedundancySupport (RedundancyMode was removed) - Integration tests: add RestrictForLegacyTfm + CustomTestTarget fallback so legacy TFM CI passes no-op instead of failing with empty TargetFrameworks - Integration Maintenance assertion updated to spec-aligned behavior (Maintenance with an available peer warrants failover, Part 4 Table 105) Validated: full UA.slnx builds on net10 + net48; Redundancy 178, Crdt 30, K8s 27, Integration 3, Client redundancy 76, Sessions failover 4 - all pass.
The private ManagedSession constructor gained enableTokenReuseFailover (bool) and networkRedundancy (NetworkRedundancyOptions?) parameters during the network-redundancy work, but two reflection-based test helpers still used the old signature, so GetConstructor returned null and failed the fixture SetUp (52 cascading failures = the test-ubuntu-latest-Client / Fast PR test CI failures). - ManagedSessionComplianceTests.CreateManagedSessionWithInner: add the 4th bool + NetworkRedundancyOptions to the ctor type list and invoke args - ManagedSessionTests: add NetworkRedundancyOptions to the ctor type list and invoke args Validated: full Opc.Ua.Client.Tests now 1530 passed / 0 failed on net10.
The legacy-TFM no-op shell strips all references, so on netstandard2.0/2.1 there is no
assembly providing System.Object and the empty compile failed with CS8021 ("No value
for RuntimeMetadataVersion"). net4x builds got System.Object from the implicit targeting
pack so they worked. Supply an explicit RuntimeMetadataVersion for the no-op shell so the
empty assembly compiles on every legacy TFM.
This was the netstandard2.0 leg of the build-*-all-tfm failure; it affected every
RestrictForLegacyTfm project (Core.Diagnostics, Network.Fuzz*, Redundancy.K8s*,
Redundancy.Integration.Tests, RedundantServer, RedundantClient, Mcp, Minimal* samples).
Validated: full UA.slnx builds on net10, net48, and netstandard2.0; the no-op shell
compiles cleanly on net472/net48/netstandard2.0/netstandard2.1.
… from codecov The new redundancy libraries are ~80% unit-covered, but codecov/patch was dragged down by genuinely-untestable Kubernetes IO (real HTTP to the API server, the readiness HTTP listener, and cluster-bound startup tasks). - Add KubernetesServerBuilderExtensionsTests: a DI-registration test that exercises UseKubernetes / UseKubernetesLeaderElection / UseKubernetesPeerDiscovery / UseKubernetesReadiness (the previously 0%-covered builder wiring) plus null-builder guards, resolving the registered services out-of-cluster with no IO. - codecov.yml: ignore the four integration-only K8s IO files (KubernetesHttpApiClient, KubernetesReadinessServer, KubernetesReadinessStartupTask, KubernetesPeerDiscoveryStartupTask) - mirrors the existing Applications/** exemption; the Kubernetes logic (models, factory, lease election, peer parsing, readiness mapping, builder wiring) stays measured. Measured redundancy-library line coverage (codecov view) is now ~88%, above the 80% patch target. K8s tests: 29 passed.
…Lock ambiguity) The net10-only RedundantServer sample used RestrictForLegacyTfm, which only no-ops legacy TFMs (net4x/ns2.x). Under the Linux all-TFM CI leg (CustomTestTarget=net8.0) the app still built net10 while referencing the net8-built Opc.Ua.Types, so its System.Threading.Lock polyfill collided with net10's BCL Lock (CS0433) in HaSampleNodeManager. Follow CustomTestTarget like the libraries so the app's framework references match the TFM being built (net8 app + net8 types => only the polyfill Lock). Validated: RedundantServer builds on net8.0 and net10.0.
Core.Diagnostics.Tests builds net8/net9/net10 and had an unconditional build-ordering ProjectReference to the net10-only McpServer (so its assembly exists for the reflective McpServer tests). On the Linux all-TFM CI legs (CustomTestTarget=net8.0/net9.0) the solution build forced McpServer to the leg TFM, which it cannot target (its own deps build net8/net9), failing the UA.slnx build. - Reference McpServer only when building net10 (CustomTestTarget '' or net10.0). - Make both reflective LoadMcpAssembly helpers Assert.Ignore (skip) when the net10 Opc.Ua.Mcp assembly is absent, instead of asserting it exists - so the net8/net9 test legs skip the MCP reflective tests cleanly while net10 still runs them. Pre-existing infra issue (unrelated to the redundancy feature) surfaced once the RedundantServer net8 Lock fix let the Linux all-TFM build progress past net8. Validated: full UA.slnx builds on net8.0/net9.0/net10.0; Core.Diagnostics McpServer tests pass on net10 (13) and skip/pass without failure on net8.
Merge Opc.Ua.Redundancy.Server.Crdt.Tests into Opc.Ua.Redundancy.Server.Tests (208 tests, 39s, well under 20min); remove the empty Opc.Ua.Redundancy.Server.Integration.Tests project (no integration tests exist); rename Opc.Ua.Client.Redundancy.Tests to Opc.Ua.Redundancy.Client.Tests. Full UA.slnx builds net10; Server 208, Client 6 pass.
New Opc.Ua.Redundancy library hosts the canonical ByteStringCrdtSerializer + CrdtSharedKeyValueStore (namespace Opc.Ua.Redundancy, shared with the Core seams). Opc.Ua.Redundancy.Server and .Client both link it; their duplicate CRDT copies removed. Adds AddCrdtClientSharedStore DI extension in Opc.Ua.Redundancy.Client (server CRDT DI already via ReplicatedServerBuilderExtensions). Full UA.slnx builds net10+net48; Server 208, Client 6 tests pass.
…phase 1) Adds the strongly-consistent consensus seam IRaftConsensus (modeled 1:1 on the RaftNode facade of marcschier/raft-cs) plus a deterministic in-process backend (InProcessRaftConsensus + InProcessRaftCluster) with a single shared totally-ordered committed log and lowest-live-id leadership. This is the offline/test backend behind the upcoming RaftSharedKeyValueStore and RaftLeaderElection; the external multi-node RaftCs engine binds to the same contract later.
…erElection (Raft phase 2) RaftSharedKeyValueStore is a replicated state machine over IRaftConsensus: Set/Delete/CAS commands are proposed, applied in committed log order against a materialized map (CAS decided deterministically at apply time), and correlated back to the caller by request id. Provides a real linearizable CompareAndSwapAsync and a WatchAsync change-feed (derived from the commit stream), plus TryGet/Scan from the materialized snapshot. RaftLeaderElection delegates ILeaderElection to native Raft leadership (no lease CAS, no split-brain). 19 unit tests incl. exactly-one-CAS-winner under contention and two-replica convergence over a shared cluster.
…cyMode (Raft phase 3)
RedundancyConsistencyMode { Eventual (default) | Strong }. HybridSharedKeyValueStore implements the Eventual mode: each key lives in exactly one backend by prefix - strong prefixes (nonce/, lease/, election/, configurable via ArrayOf<string>) route entirely to the linearizable Raft store; all other keys route to the eventually-consistent CRDT store. A spanning/empty-prefix scan merges both. 8 routing unit tests.
…onsistency (Raft phase 4) Adds RedundancyConsistencyOptions + UseRedundancyConsistency(builder) which registers a shared IRaftConsensus singleton, the mode-appropriate ISharedKeyValueStore (Strong -> RaftSharedKeyValueStore; Eventual -> HybridSharedKeyValueStore over a CRDT/InMemory bulk + Raft strong store), and a native RaftLeaderElection. Uses TryAddSingleton so it composes before UseDistributedAddressSpace/UseDistributedSessions and wins. 4 DI tests.
SharedSingleUseNonceRegistry and SharedStoreLeaseElection already consume ISharedKeyValueStore.CompareAndSwapAsync, so registering a Raft/Hybrid store (phase 4) makes them linearizable with no code change. Tests prove: exactly-once nonce under contention over Raft; single-leader lease under contention; and that UseRedundancyConsistency auto-wires a CAS-capable store for the CRDT session factory's nonce registry (no separate Redis backend required).
…re (Raft phase 6) AddRaftClientSharedStore registers a Raft-backed ISharedKeyValueStore + native RaftLeaderElection sharing one IRaftConsensus, giving a client replica set linearizable shared state and primary-client election in one call (the ClientReplicaCoordinator consumes both). AddRedundantClientSharedStore(mode,...) is mode-aware: Strong -> Raft store; Eventual -> Hybrid over a CRDT bulk + Raft strong store. 4 client DI tests.
…s (Raft phase 7, gated) Adds the RaftCsConsensus : IRaftConsensus adapter that wraps a RaftCs RaftNode (ProposeAsync, Committed apply-stream, IsLeader -> LeadershipChanged poll). Compiled out behind #if OPCUA_RAFTCS so the repo builds/tests fully offline against InProcessRaftConsensus. The file header documents the activation steps (add RaftCs/RaftCs.Transport[.NanoMsg] package refs from Crdt 1.0.5, define OPCUA_RAFTCS, wire via RaftConsensusFactory). No package references added yet - deferred until the user signals the nuget.org pull.
HighAvailability.md gains a 'Consistency modes' section (Strong Raft vs Eventual CRDT-complemented-by-Raft, UseRedundancyConsistency, RaftLeaderElection, client extensions, RaftCs/k8s notes) and the stale 'keep nonce on a strongly consistent store (e.g. Redis)' caveat now points at the in-package Raft layer. HighAvailabilityKubernetes.md notes the native Raft election + StatefulSet/quorum/WAL guidance. NugetREADME for Opc.Ua.Redundancy lists the CP/AP/Hybrid building blocks.
…ft consensus Bumps Crdt/Crdt.Transport 1.0.2 -> 1.1.0 and adds RaftCs, RaftCs.Transport (+ .Transport.NanoMsg, .Storage.File) 1.1.0 to central package management. Opc.Ua.Redundancy now references RaftCs + RaftCs.Transport and defines OPCUA_RAFTCS, activating the RaftCsConsensus adapter (which compiled 1:1 against the real RaftNode API). Adds RaftCsConsensus.CreateSingleNode() (a real single-voter RaftCs replica with in-memory storage/transport that self-elects) plus a bounded wait-for-initial-leader so the first proposals are not dropped during election, and owned-host disposal for the in-memory network. Wires RaftCs as the DI default IRaftConsensus in UseRedundancyConsistency (server) and AddRaftClientSharedStore/AddRedundantClientSharedStore (client); InProcessRaftConsensus remains the lightweight deterministic default for the parameterless store ctor and direct/unit use, and RaftConsensusFactory plugs in a multi-node RaftNode. Validation: Server.Tests 246/246 + Client.Tests 10/10 on net10 AND net48; new RaftCsConsensusTests exercise election/CAS/nonce over the real engine; a NativeAOT publish of Opc.Ua.Aot.Tests runs 2 new RaftAotTests green (RaftCs is trim/AOT-clean). Docs/HighAvailability.md + NugetREADME updated to reflect the activated default.
…fallback (review) Code-review follow-ups (security + data-consistency were clean): HIGH: RaftSharedKeyValueStore proposals could hang forever in a multi-pod no-leader / leadership-change / lost-quorum window (RaftCs drops a follower proposal when LeaderId==0, and the commit signal is only the Committed stream). Add a bounded commit timeout (default 30s, configurable; Timeout.InfiniteTimeSpan to opt out) that fails the pending proposal with TimeoutException, and fail all pending proposals if the single applier ever dies. The single-node default was already safe. MEDIUM: In Eventual mode the distributed address-space change-feed routed to a CRDT bulk store whose WatchAsync throws NotSupported, silently faulting the standby's topology-sync task. InMemoryNodeStateStore.SubscribeChangesAsync now catches NotSupportedException and falls back to periodic scan-polling (configurable interval, default 2s) so standby sync keeps working over any store. LOW: the UseRedundancyConsistency-before-UseDistributed* ordering requirement remains documented (TryAddSingleton). Tests: ProposalTimesOutWhenNoCommitOccurs + SubscribePollsWhenStoreHasNoWatch; full Server.Tests 248/248 on net10 AND net48.
…y + 3-node tests (rw-A1/B) CreateCluster builds a multi-node RaftCs replica over static ConfState membership with an injectable IRaftTransport + IRaftWritableStorage (so Opc.Ua.Redundancy stays NanoMsg/File-free and AOT-clean); a memberIds overload uses in-memory MemoryStorage, a storage overload takes a durable WAL, and CreateSingleNode is refactored to a 1-member special case. RaftCsClusterTests spin up a real 3-node cluster over RaftCs InMemoryNetwork: follower-write convergence (leader forwarding), leader failover + re-election stays writable, and quorum-loss makes a proposal fail via the commit timeout. 7 tests pass in ~1s.
… (rw-A2) Opc.Ua.Redundancy.K8s now references RaftCs.Transport.NanoMsg + RaftCs.Storage.File (AOT-clean spike: builds 0-warning with IsAotCompatible; NanoMsgSharp is managed). UseKubernetesRaftConsensus derives this replica's Raft node id from the StatefulSet ordinal, builds member ids from ReplicaCount (static membership), wires a NanoMsgBusTransport (bind + headless-DNS peers) and a crash-safe FileRaftStorage on a PersistentVolume path (in-memory optional; a fresh WAL is bootstrapped with the static ConfState), and registers the result via RaftCsConsensus.CreateCluster as IRaftConsensus so UseRedundancyConsistency composes over it. 3 registration tests.
…tServer (rw-C) Adds HA_CONSISTENCY=strong to the RedundantServer sample: it registers UseRedundancyConsistency(Strong) backed by a multi-node RaftCs cluster over NanoMsg (HA_RAFT_ID/MEMBERS/BIND/PEERS), so the shared store, leader election, and session nonce are linearizable and shared across containers - the real cross-container HA the in-memory active/passive compose lacks. Adds docker-compose.raft.yml (a runnable 3-node Raft cluster) and documents HA_CONSISTENCY + HA_RAFT_* and the Raft-vs-Redis alternative in the README. Sample builds AOT-clean (0 warnings).
… multi-node wiring (rw-A3) HighAvailability.md now points at RaftCsConsensus.CreateCluster + UseKubernetesRaftConsensus (and the RedundantServer HA_CONSISTENCY sample) for multi-pod; HighAvailabilityKubernetes.md documents UseKubernetesRaftConsensus (ordinal->id, headless-DNS peers, File WAL on a PVC) with a code example; the K8s NugetREADME lists the new extension.
Coverage pass: all new Raft library files are >=80% from Server.Tests (RaftLeaderElection 83%, RaftCsConsensus 88%, RaftSharedKeyValueStore 92%, Hybrid 93%, InProcessRaftConsensus 95%, cluster/options/builder 96-100%). Adds a K8s test exercising the previously-untested UseDurableStorage=true branch (FileRaftStorage on a temp dir + fresh-WAL ConfState bootstrap) with fully-qualified peer DNS.
Resolved conflicts: UA.slnx (kept both feature Redundancy projects and upstream PubSub/Core.Schema.Tests projects); CryptoUtils.cs (took upstream canonical ZeroMemory/FixedTimeEquals with the broader NETSTANDARD2_1_OR_GREATER||NET5_0_OR_GREATER guard); ManagedSessionBuilder.cs (combined upstream BuildChannelBindings/WithDefaultTcp + the new WebApi endpoint branch with the feature's ownedHttpClientFactory try/finally disposal safety).
- Session: move browse/history continuation-point bookkeeping (store, lists, mirrored-owner maps) into a focused SessionContinuationPoints class; Session delegates via thin forwarders (review t1). - Subscription: move sent-message ring, sequence-number counter and retransmission-store plumbing into a focused SentMessageQueue class, keeping Publish/Acknowledge/Republish focused on orchestration (review t2). - Rename Docs/HighAvailabilityKubernetes.md to Docs/Kubernetes.md and update references (review t3). - Update SubscriptionLifecycleTests reflection helper to reach m_sentMessages via m_messageQueue. Behavior preserved (verbatim logic moved); validated: Subscriptions 505, Redundancy.Server 251, Server 1895, Sessions continuation-point 29 pass; Opc.Ua.Server builds clean net10+net48.
…aining-work file Plans 28-32 were almost entirely delivered (CRDT + Raft in-package, secure session mirroring, transparent/non-transparent redundancy, samples, docs). Replace the five superseded plan files with one plans/28-distributed-ha-remaining.md that lists only the not-yet-delivered work: async ISubscriptionStore, transparent-redundancy worked sample, and large-address-space hydration optimization.
| return; | ||
| } | ||
|
|
||
| // A single ManagedSession is the managed client. WithServerRedundancy() lets it |
There was a problem hiding this comment.
Remove "managed client" wording. Key point: "Create a normal managed session, opt it into Server redundancy handling, thats it"
| session.ConnectionStateChanged += OnConnectionStateChanged; | ||
|
|
||
| await LogRedundancyInfoAsync(session, ct).ConfigureAwait(false); | ||
| await SubscribeToCurrentTimeAsync(session, ct).ConfigureAwait(false); |
There was a problem hiding this comment.
Should also subscribe to a "Replicated" value (from ha node manager in redundantServer sample).
| } | ||
| } | ||
|
|
||
| private static async Task RunReplicaSetAsync( |
There was a problem hiding this comment.
Maybe this can be generalized in Opc.Ua.Client or Opc.Ua.Redundancy.Client to be set up using DI (key store, etc.) and that makes for a simpler setup for users that want to use client redundancy? Ideally the user just gets the shape of ISession to worry about and the redundancy is "transparent" (which hides all coordination and sessions that need to be managed). If the "session" was built with > 1 replicas. Whereby if it was built with 0/1 replicas it just defaults to the current implementation and user gets just a ManagedSession (ISession) just like today. Then above would simplify as the setup of the session and subscriptions are the same.
| @@ -0,0 +1,91 @@ | |||
| # Strongly-consistent (Raft) active/passive redundant OPC UA servers plus a managed client. | |||
| # | |||
| # Unlike docker-compose.active-passive.yml (whose default in-memory store is private to each | |||
There was a problem hiding this comment.
Make this the real active-passive.yml and remove the "existing active/passive yml".
| @@ -0,0 +1,63 @@ | |||
| # Active/active redundant OPC UA servers (CRDT gossip) plus a managed client. | |||
| # | |||
There was a problem hiding this comment.
Make eventual consistency/strong consistency configurable via env var.
IF possible, make a single docker-compose.yml that is env configurable to run as
- a/a eventual (default),
- a/a strong,
- a/p
IF possible also make the number of replicas configurable
| # Active/passive: leader-election wiring demonstration, plus a client. | ||
| docker compose -f Applications\RedundantServer\docker-compose.active-passive.yml up --build | ||
|
|
||
| # Strong consistency: a real 3-node RaftCs cluster (shared store across containers), plus a client. |
There was a problem hiding this comment.
Add docker-compose.yml to the client as well to run either single, or multi replica clients against the server. Determine a good way to show the replication working, e.g. by scaling up the number of clients to saturate a single replica, etc.
|
|
||
| The API names distinguish standardized OPC UA model wiring from deployment extensions. `AddServerRedundancy(...)`, `AddServerServiceLevel(...)`, and `AddRequestServerStateChange(...)` publish or maintain OPC 10000-4 §6.6 nodes/methods; `UseDistributedAddressSpace(...)`, `UseDistributedSessions(...)`, `UseDistributedSubscriptionMirroring(...)`, `UseKubernetesLeaderElection(...)`, `UseKubernetesPeerDiscovery(...)`, and `UseKubernetesReadiness(...)` register beyond-spec extension services. `AddServerRedundancy(...)` does not drive `Server.ServiceLevel` by itself, so Kubernetes readiness must also register a ServiceLevel provider, commonly with `AddServerServiceLevel(...)` or the leader-aware provider used by the sample. | ||
|
|
||
| ## Redundancy shapes on Kubernetes |
There was a problem hiding this comment.
Also document the setup on Kubernetes that does not require any K8s integration (due to raft/crdt use)
| edits via a code-fixer. | ||
| - Open an issue on | ||
| [OPCFoundation/UA-.NETStandard](https://github.com/OPCFoundation/UA-.NETStandard/issues). | ||
|
|
There was a problem hiding this comment.
Remove, new and not breaking change
| Documented limitations: | ||
|
|
||
| - The synchronous core `ISubscriptionStore` definition-persistence contract requires a synchronously-completing backend such as the in-memory or CRDT store. Full async persistence to a backend such as Redis would require an async `ISubscriptionStore`. | ||
| - `SharedKeyValueSubscriptionStore` restores definitions and retransmission state, but monitored-item data/event queues are not restored by `RestoreDataChangeMonitoredItemQueue` or `RestoreEventMonitoredItemQueue`. | ||
| - Continuation-point mirroring is best-effort. Built-in node-manager `ContinuationPoint.Data` is opaque and is not reconstructed on a backup; after failover a client may receive `BadContinuationPointInvalid` and re-issue Browse or HistoryRead, which OPC 10000-4 §6.6.2.2 permits. Node managers that can serialize their own continuation-point data may opt in through `IContinuationPointStore`. | ||
| - Deterministic EventIds are optional and only as stable as the event fields used. Alarms & Conditions clients should still call `ConditionRefresh` after failover as required by OPC UA. |
There was a problem hiding this comment.
Plan addressing the limitations
| ct); | ||
| ``` | ||
|
|
||
| OPC UA does not standardize how active and backup clients exchange `SessionId` or subscription ids. The coordinator uses `ServerDiagnostics` when authorized; deployments that disable diagnostics should provide their own coordination channel. |
There was a problem hiding this comment.
Lets not leave this vague, this should be using the Raft/Crdt kv registered, not server diagnostics as it requires admin rights.
|
|
||
| `UseRedundancyConsistency` registers a shared `IRaftConsensus` plus a native `RaftLeaderElection` (`ILeaderElection`). Raft leadership is decided by the consensus protocol itself — a single leader per term, no split-brain — which is stronger than the lease-CAS `SharedStoreLeaseElection`. Because the distributed features register their store and election with `TryAddSingleton`, call `UseRedundancyConsistency` first and they compose over the chosen store and election. | ||
|
|
||
| The consensus engine is pluggable through the `IRaftConsensus` seam. By default the DI registration uses a single-node [`RaftCs`](https://github.com/marcschier/raft-cs) replica (`RaftCsConsensus.CreateSingleNode()`, a real RaftNode with in-memory storage/transport that elects itself leader); `InProcessRaftConsensus` is a lighter deterministic in-process alternative for tests and in-process replica sets. For a multi-pod cluster, build a multi-node replica with `RaftCsConsensus.CreateCluster(nodeId, memberIds, transport, …)` — a `RaftCs.Transport.NanoMsg` transport (peers addressed by DNS) plus an in-memory or durable `RaftCs.Storage.File` WAL — and register it through `RaftConsensusFactory`. On Kubernetes, `UseKubernetesRaftConsensus` (in `Opc.Ua.Redundancy.K8s`) does this for you from the StatefulSet ordinal and headless-Service DNS. RaftCs ships its own `IRaftTransport` (NanoMsg), so Raft shares the NanoMsg substrate with the CRDT gossip layer. The `Applications/RedundantServer` sample (`HA_CONSISTENCY=strong`, `docker-compose.raft.yml`) is a runnable multi-node example. |
There was a problem hiding this comment.
Remove [RaftCs](https://github.com/marcschier/raft-cs)
…rSet Adds an opt-in, gated server-side load-direction layer that answers a GetEndpoints request on a dedicated balancing discovery URL with the best peer's endpoints, complementing (never replacing) the standard client-driven RedundantServerArray/ServiceLevel selection. - Core seam: IGetEndpointsDirector + StandardServer.GetEndpointsDirector property + a guarded hook in GetEndpointsAsync + hosted-service wiring (additive; default null = unchanged behavior). - Opc.Ua.Redundancy.Server/Redundancy/LoadDirection: health ServiceLevel + separate load-weight gossip (publish/view with staleness age-out, fail-to-self, integrity/malformed drop, coalesced load), integrity-protected peer endpoint directory, BandedServerDirectionPolicy (health-tier -> least-load -> random), gated ServerLoadDirector (publish-own-on-normal-URL, redirect-on-balancing-URL, fail-safe), LoadDirectionStartupTask, UseServerLoadDirection DI, thread-safe tie-break. - ServiceLevel stays a pure health/eligibility signal; a separate load weight drives A/A balancing. Consistency follows UseRedundancyConsistency; the high-churn load weight is always coalesced. - Records are integrity-protected via IRecordProtector (verify-before-use, fail-closed). - Docs: HighAvailability.md 'GetEndpoints load direction' section (usage, conformance caveats, consistency, security/threat model, pitfalls). - 24 tests (view/endpoints/policy/director). Builds clean net10 + net48 (0 warnings); full Redundancy.Server.Tests suite 275 pass.
…/R2/R4) - R2: LoadDirectionOptions.StrongEligibility routes the eligibility keyspaces (svc/ + endpoint/) to the linearizable Raft store in hybrid mode via a composable IStrongKeyspaceProvider that UseRedundancyConsistency aggregates; the high-churn load/ weight stays eventual. Hybrid store exposes IsStrongKey + DefaultStrongKeyPrefixes. + 2 routing tests. - R1: LoadDirectionServerIntegrationTests drives a real StandardServer via ServerFixture with a configured ServerLoadDirector and a seeded peer; asserts GetEndpoints on the balancing URL redirects to the peer and a normal URL serves local. Validates the StandardServer.GetEndpointsAsync seam end-to-end. + 2 tests. - R4: Applications/RedundantServer wires UseServerLoadDirection from HA_BALANCING_URL (StrongEligibility when HA_CONSISTENCY=strong); adds docker-compose.loaddirection.yml + README docs. Full Redundancy.Server.Tests suite 279 pass; builds clean net10 + net48 (0 warnings).
Convert the subscription-definition lifecycle methods on ISubscriptionStore to async so definitions can be persisted to an async network backend without a sync-over-async wrapper: - StoreSubscriptions -> ValueTask<bool> StoreSubscriptionsAsync(.., CancellationToken) - RestoreSubscriptions -> ValueTask<RestoreSubscriptionResult> RestoreSubscriptionsAsync(CancellationToken) - OnSubscriptionRestoreComplete -> ValueTask OnSubscriptionRestoreCompleteAsync(.., CancellationToken) The per-monitored-item queue-restore hooks (RestoreDataChangeMonitoredItemQueue, RestoreEventMonitoredItemQueue) stay synchronous because they run on the synchronous monitored-item creation path. SubscriptionManager awaits the new methods. SharedKeyValueSubscriptionStore now awaits its shared-store writes directly (removes the fire-and-forget Complete/ Observe helpers). The Quickstarts DurableSubscription sample and the store/AOT tests are updated to the async signatures. Docs (HighAvailability, migration guide, remaining-work plan) updated. Validated: net10 + net48 clean (0 warnings on changed libs); 279 Redundancy.Server + 145 Server subscription + 23 store tests pass.
Extend Applications/RedundantServer with a worked transparent-redundancy deployment: two REDUNDANCY_MODE=transparent replicas that present ONE logical OPC UA server behind ONE virtual endpoint. - Program.cs: add HA_APPLICATION_URI (shared ApplicationUri - CreateSession validates the client serverUri against it), HA_SUBJECT_NAME + HA_PKI_ROOT (shared ApplicationInstanceCertificate) so all replicas present one identity. - docker-compose.transparent.yml + nginx.transparent.conf: two replicas sharing one URI + one certificate (seeded into a shared PKI volume by server-a, reused by server-b) mirroring session + address-space state by CRDT gossip, behind an nginx TCP load balancer publishing the single virtual endpoint. Each replica advertises the load-balancer host (HA_HOST=lb); because that host is a DNS name the listener binds to all interfaces while discovery/CreateSession echo the one endpoint the client uses (ServerBase.FilterByEndpointUrl match). - Docs: RedundantServer/README.md transparent section + settings; Docs/ HighAvailability.md worked-sample pointer. Compose validated with 'docker compose config'; sample builds clean.
Add INodeStateStore.EnumerateValuesAsync so a standby hydrates its local graph in two streamed passes - EnumerateAsync for node topology, then EnumerateValuesAsync for the latest variable values - instead of one TryReadValueAsync round trip per variable. AddressSpaceSynchronizer.SeedOrHydrateAsync now applies values from the single streamed value pass, so hydrating a large address space costs a bounded number of round trips against a networked (CRDT/Raft) backend rather than O(variables). InMemoryNodeStateStore implements it by scanning the value keyspace (same pattern as EnumerateAsync over the node keyspace). Node topology is still materialized eagerly; snapshot/lazy topology materialization for very large graphs remains a documented future optimization (the CRDT path already exchanges a snapshot + deltas). 3 store tests added. net10 + net48 + AOT build clean; 282 Redundancy.Server tests pass.
| * http://opcfoundation.org/License/MIT/1.00/ | ||
| * ======================================================================*/ | ||
|
|
||
| #nullable enable |
There was a problem hiding this comment.
Remove all #nullable enable from files in projects that are already <nullable>enable in their .csproj file
There was a problem hiding this comment.
Rename back to .Kubernetes instead of .K8s (folder and project name and namespaces)
| using System.Runtime.CompilerServices; | ||
|
|
||
| [assembly: CLSCompliant(false)] | ||
| [assembly: InternalsVisibleTo("Opc.Ua.Aot.Tests")] |
There was a problem hiding this comment.
Use the .csproj method <InternalsVisibleTo> like in all other projects (change everywhere assembly attributes are used for it)
| /// to every node manager that opts in via <see cref="ILocalAddressSpaceSource"/>, | ||
| /// enabling active/active (multi-writer) replication of its address space. | ||
| /// </summary> | ||
| public sealed class CrdtAddressSpaceStartupTask : IServerStartupTask, IAsyncDisposable |
There was a problem hiding this comment.
Where possible change "Crdt" prefix of classes to "Replicated" across the PR. If there are conflicts use "CrdtReplicated" as prefix. It works nicer with the options name
| throw new ArgumentNullException(nameof(eventState)); | ||
| } | ||
|
|
||
| var builder = new StringBuilder(); |
There was a problem hiding this comment.
Maybe better to binary serialize the nodestate and compute the Hash? Also, maybe use a faster hashing algorithm (e.g. not crypto)? Do some benchmarks.
| // =========================================================================== | ||
| // Adapter that binds the external Raft engine `RaftCs` | ||
| // (https://github.com/marcschier/raft-cs, shipped alongside the Crdt 1.1.0 | ||
| // libraries) to the in-repo IRaftConsensus seam. Opc.Ua.Redundancy references | ||
| // the `RaftCs` and `RaftCs.Transport` packages and defines OPCUA_RAFTCS, so | ||
| // this type is compiled into the assembly. The guard remains so the file can | ||
| // be excluded if the RaftCs dependency is ever removed. | ||
| // | ||
| // Usage: | ||
| // - Single-node / in-process: RaftCsConsensus.CreateSingleNode(). | ||
| // - Multi-pod: construct a RaftNode with durable storage | ||
| // (RaftCs.Storage.File) and a networked transport | ||
| // (RaftCs.Transport.NanoMsg, sharing the NanoMsg substrate with the CRDT | ||
| // gossip layer), then `new RaftCsConsensus(node)`. Wire either through | ||
| // RedundancyConsistencyOptions.RaftConsensusFactory (server) or the client | ||
| // AddRaftClientSharedStore/AddRedundantClientSharedStore factories. | ||
| // =========================================================================== | ||
|
|
||
| #if OPCUA_RAFTCS |
There was a problem hiding this comment.
Remove including guard ifdef. Rename class to "DefaultRaftConsensus"
Phase 1-2 of snapshot+delta address-space hydration (fast time-to-ready). - NodeStateChange gains a monotonic Sequence. - InMemoryNodeStateStore embeds a single-writer monotonic sequence inside each protected n/ and v/ record; every read path parses it and CurrentSequence / ObserveSequence expose the high-water mark (reseed-on-promotion). - New INodeStateSnapshotStore capability (optional; store falls back to the streamed path when absent): WriteSnapshotAsync builds a chunked, atomically published snapshot (snap/ + snapmeta/manifest, 1 MB chunks, predecessor-gen GC) and trims the bounded delta log; TryReadSnapshotAsync streams it back; every node/value/delete write appends dlog/<seq>, and ReadDeltaLogAsync replays entries after a sequence in order. - InMemoryNodeStateStore is now IDisposable (snapshot semaphore); the registry already disposes registered stores. net10 + net48 clean (0 warnings); 287 Redundancy.Server tests pass (5 new store tests: snapshot round-trip, delta-log replay, trim, sequence high-water).
Summary
This branch implements OPC 10000-4 §6.6 Redundancy end to end — server and client, transparent and non-transparent, across all failover modes (
None,Cold,Warm,Hot,HotAndMirrored,Transparent) — and layers an opt-in distributed high-availability (HA) capability on top so replicas can share address-space, session, and subscription state and expose redundancy to clients through the documented OPC UA mechanisms (for example, running a server replica set across nodes in Kubernetes).The single-instance, in-memory path stays zero-overhead: every distributed feature is opt-in through dependency injection, with a direct-construction fallback, and replaced APIs are kept and marked
[Obsolete].New packages
OPCFoundation.NetStandard.Opc.Ua.Server.RedundancyOPCFoundation.NetStandard.Opc.Ua.Server.Redundancy.CrdtOPCFoundation.NetStandard.Opc.Ua.Server.Redundancy.K8sAlso adds the worked samples
Applications/RedundantServerandApplications/RedundantClient, and the guidesDocs/HighAvailability.mdandDocs/HighAvailabilityKubernetes.md.Server side (§6.6)
AddServerRedundancy(...)populates the liveServer.ServerRedundancynodes (RedundancySupportplusRedundantServerArray/ServerUriArray/CurrentServerId) and drivesServer.ServiceLevelfrom anIServiceLevelProvider(leader high, standby low).Server.ServerRedundancyis typed to the NonTransparent or Transparent subtype according to the configured mode.AddRequestServerStateChange(...)implements the §6.6.5 manual-failover method, withEstimatedReturnTimeand the Maintenance / NoDataServiceLevelsub-ranges.FindServers(ConfiguredRedundantServerSetProvider) and advertise theNTRSdiscovery capability for GDS/NTRS registration.EventIdsynchronization for Transparent and HotAndMirrored sets (§6.6.2.2), excluding per-replica fields so de-duplicating clients do not double-process events.Client side (§6.6)
DefaultServerRedundancyHandlerreadsServer.ServerRedundancyandServer.ServiceLeveland fails over to the highest-ServiceLevel peer. It honors Maintenance (Part 4 Table 105 — disconnect to an available peer), keeps Warm backups disabled until failover (Table 107), and treats peers known only throughServerUriArrayas failover candidates.RedundantManagedClientimplements the Cold, Warm, Hot (a), Hot (b), and HotAndMirrored client patterns: one active session, optional lightweight ServiceLevel status-check sessions to backups, and — for Hot (b) — merging of concurrent reporting streams with exact value-identity de-duplication. HotAndMirrored failover re-activates the mirrored session with the existingAuthenticationTokeninstead of recreating it.Distributed state (opt-in; an extension beyond §6.6)
ISharedKeyValueStore(in-memory default) andINodeStateStore, with anAddressSpaceSynchronizerthat bridges aCustomNodeManager2's predefined nodes to the shared store, so node and reference additions/removals and value changes propagate to other replicas.IDistributedValueCachelets read/write callbacks cache the last value with a freshness bound; monitored items use the normal read pipeline and therefore participate only when the read path does.DistributedSessionManager(wired through a new additiveISessionManagerFactoryseam onStandardServer) mirrors encrypted session state across replicas. On a failover reconnect the standby restores the session and performs the fullActivateSessionclient-certificate signature validation against a single-useserverNonceconsumed via compare-and-swap across the replica set, enforcing the same SecurityPolicy and SecurityMode. TheAuthenticationTokenis a lookup key only, never an authenticator: entries are keyed by the SHA-256 digest of the token, and a cross-replica restore emits a distinct audit event.IRecordProtector; secret-bearing mirrors require a registered protector or an explicit opt-out.Active/active (CRDT)
Opc.Ua.Server.Redundancy.Crdtadds leaderless multi-writer address-space replication with CRDTs and gossip (UseReplicatedAddressSpace) plus CRDT-backed session metadata (UseReplicatedSessions). Networked gossip is fail-closed without authenticated transport (mutual TLS for TCP; an explicit opt-out for isolated dev/test). CRDT state is eventually consistent, so exactly-once decisions (such as the single-use nonce registry) stay on a strongly consistent store. The package is available on all stack target frameworks.Kubernetes
Opc.Ua.Server.Redundancy.K8sprovides Kubernetes Lease leader election, EndpointSlice-based peer discovery, and ServiceLevel-driven readiness for running an OPC UA server replica set.Docs/HighAvailabilityKubernetes.mdcovers StatefulSet/Deployment and Service manifests, RBAC, probes, time synchronization, secrets, gossip-port NetworkPolicy, and GDS/NTRS registration.Compatibility and validation
ISessionManagerFactoryandIServerStartupTask); replaced APIs marked[Obsolete]for migration.net472,net48,netstandard2.0,netstandard2.1,net8.0,net9.0,net10.0); NativeAOT-compatible where the runtime supports it.See
Docs/HighAvailability.mdfor the full OPC 10000-4 §6.6 mapping, theAdd*(standard nodes) versusUse*(extension) builder convention, and the security/rotation guidance.Update — package consolidation + strong consistency (Raft)
Since the summary above was written, the redundancy packages were consolidated and renamed, client redundancy was folded into the managed session, and a strongly-consistent (Raft) layer was added. The branch is also merged up to the latest
master.Packages (current):
Opc.Ua.Redundancy(shared CRDT + Raft building blocks),Opc.Ua.Redundancy.Server,Opc.Ua.Redundancy.Client,Opc.Ua.Redundancy.K8s. (The former*.Server.Redundancy/.Redundancy.Crdt/.Redundancy.K8snames are replaced; the CRDT active/active code merged into the server package.)Client:
RedundantManagedClientis removed — redundancy is transparent and built intoManagedSession(viaIManagedSession/ManagedSessionBuilder), so connecting to a non-redundant, redundant (transparent or non-transparent), or client-replica-set server needs no special API.Strong consistency (Raft), an extension beyond §6.6. A CP Raft layer complements the AP CRDT store behind the same
ISharedKeyValueStoreseam, selectable withRedundancyConsistencyMode(Eventualdefault,Strong) viaUseRedundancyConsistency:RaftSharedKeyValueStoreprovides a real linearizableCompareAndSwapAsync+WatchAsync;RaftLeaderElectiongives native single-leader election (no split-brain);HybridSharedKeyValueStoreroutes the strong keyspaces (single-use nonce, lease, election) to Raft and bulk state to the CRDT store.IRaftConsensus:InProcessRaftConsensusfor single-process/tests, and the externalRaftCsengine (marcschier/raft-cs, over NanoMsg with an optional file WAL) for multi-pod viaRaftCsConsensus.CreateCluster/UseKubernetesRaftConsensus. RaftCs is NativeAOT-clean.RedundantServersample gainsHA_CONSISTENCY=stronganddocker-compose.raft.yml(a real cross-container Raft cluster). SeeDocs/HighAvailability.md(Consistency modes).